Markov Chains and Intuitionism. III Note on Continuous Functions with an Application to Markov Chains

Continuous Time Markov Chains

Discrete-time Markov chains are useful in simulation, since updating algorithms are easier to construct in discrete steps. They can also be useful as crude models of physical, biological, and social processes. However, in the physical and biological worlds time runs continuously, and so discrete-time mathematical models are not always appropriate. This is especially true in population biology –...

Continuous-time Markov Chains

On Markov Chains with Continuous State Space

In this expository paper, we p r o ve the following theorem, which m a y be of some use in studying Markov chain Monte Carlo methods like hit and run, the Metropolis algorithm, or the Gibbs sampler. Suppose a discrete-time Markov c hain is aperiodic, irreducible, and there is a stationary probability distribution. Then from almost all starting points the distribution of the chain at time n conv...

Empirical Bayes Estimation in Nonstationary Markov chains

Estimation procedures for nonstationary Markov chains appear to be relatively sparse. This work introduces empirical  Bayes estimators  for the transition probability  matrix of a finite nonstationary  Markov chain. The data are assumed to be of  a panel study type in which each data set consists of a sequence of observations on N>=2 independent and identically dis...

Distributions of Reward Functions on Continuous-time Markov Chains

We develop algorithms for the computation of the distribution of the total reward accrued during [0, t) in a finite continuous-parameter Markov chain. During sojourns, the reward grows linearly at a rate depending on the state visited. At transitions, there can be instantaneous rewards whose values depend on the states involved in the transition. For moderate values of t, the reward distributio...